Planning with Closed-Loop Macro Actions

Planning and learning at multiple levels of tempo ral abstraction is a key problem for arti cial intelli gence In this paper we summarize an approach to this problem based on the mathematical framework of Markov decision processes and reinforcement learn ing Conventional model based reinforcement learning uses primitive actions that last one time step and that can be modeled independently of the learning agent These can be generalized to macro actions multi step actions speci ed by an arbitrary policy and a way of completing Macro actions generalize the classical no tion of a macro operator in that they are closed loop uncertain and of variable duration Macro actions are needed to represent common sense higher level actions such as going to lunch grasping an object or travel ing to a distant city This paper generalizes prior work on temporally abstract models Sutton and ex tends it from the prediction setting to include actions control and planning We de ne a semantics of mod els of macro actions that guarantees the validity of planning using such models This paper present new results in the theory of planning with macro actions and illustrates its potential advantages in a gridworld task